Event latency mitigation and screen selection

ABSTRACT

In an embodiment, the system includes a direct input device (the “client”) executing a client application (“app”), a secondary device (the “server”) executing a server app, and a network link between the server and the client. The server app transmits to the client app, via the network link, a portion of the content displayed on the server&#39;s computer screen. The client app may display this portion. Whenever the user interacts with the client device, the client app transmits those interactions to the server app over the network link. The server app sends an acknowledgement to the client app and provides the received interactions to the spatial input software executing on the server, and the spatial input software consumes the interactions as if the user performed the interactions on the server device itself.

PRIORITY CLAIM

This patent application claims the benefit of priority, under 35 U.S.C.Section 119 to U.S. Provisional Patent Application Ser. No. 62/297,035,entitled “EVENT LATENCY MITIGATION AND SCREEN SELECTION,” filed on Feb.18, 2016 to Donelli et al, which is hereby incorporated by referenceherein in its entirety.

TECHNICAL HELD

Embodiments pertain to improved synchronization between two computingdevices. Some embodiments relate to latency mitigation and screenselection when one computing device is used as an input device for asecond computing device over a computer network.

BACKGROUND

Small and portable computing devices with sensitive touch screendisplays have become ubiquitous. These devices often feature networkingcapabilities that allow it to send and receive data over one or morecomputer networks. Example networking capabilities include communicationover Wireless Local Area Networks (WLANs) such as according to an 802.11family of standards, communication over cellular networks such asaccording to a Long Term Evolution (LTE) family of standards, shortrange communications to a BLUETOOTH standard or by using Near FieldCommunications (NFC), and the like. Example computing devices includetablet computers such as an APPLE® IPAD®, a MICROSOFT® SURFACE® and thelike.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments or examples discussed inthe present document.

FIG. 1 illustrates a system for using a direct input device with spatialinput software executing on a server device, according to an embodiment.

FIG. 1B illustrates a navigator which shows a miniature replica of theentire server display, that may be displayed on the client displayaccording to an embodiment.

FIG. 2 illustrates a series of user inputs onto the client device,according to an embodiment.

FIG. 3 illustrates how input events flow within the system, according toan embodiment.

FIG. 4 is a portion of a class diagram illustrating relationshipsamongst input device classes for various pens/styli, according to anembodiment.

FIG. 5 illustrates a client event buffer being populated with inputevents generated by a stylus stroke, according to an embodiment.

FIG. 6 illustrates states of event buffers resulting from communicationsbetween a client EventStream and a server EventStream, according to anembodiment.

FIG. 7 the event rendering engine dispatching a received input event 306to one or more render units, according to an embodiment.

FIG. 8 is a block diagram illustrating an example of a machine, uponwhich any one or more embodiments may be implemented.

FIG. 9 shows an example method of a client device rendering inputaccording to an embodiment.

FIG. 10 shows an example method of changing the portion of the serverdisplay that is displayed on the client device according to anembodiment.

FIG. 11 shows a method of a server processing an input event sent from aclient device according to an embodiment.

DETAILED DESCRIPTION

With some types of software programs, pen/stylus-based input is morenatural than other forms of input. Such software programs, termed“spatial input” software, include but are not limited to: drawing, photoediting, painting, illustration, computer-aided design/drafting (“CAD”),video editing, three-dimensional (“3D”) modeling, animation, graphicdesign, industrial design, 3D sculpting, motion graphics, annotation andcollaboration, and note taking. Some existing digital pens/styli allowthe pen/stylus to be used on almost any surface (e.g., a table or desk).Such pens/styli are often referred to as “indirect input devices”because the input space (e.g., the pen/stylus on a table or desk) doesnot coincide with the display space (e.g., a computer screen). For manyusers, however, using indirect input devices with spatial input softwaremay be awkward because the user is unable to view the input space anddisplay space simultaneously. A “direct input device” is a device wherethe input space coincides with the display space. Example direct inputdevices include devices with touchscreens (e.g., tablets andsmartphones), pen displays, etc.

Users of spatial input software often prefer to use a combination of adirect input device and a secondary device; the direct input device isused for input and immediate visual output, while the secondary deviceexecutes the spatial input software. For example, the direct inputdevice may be a tablet computer with a touchscreen and the secondarydevice may be a desktop computer executing spatial input software.However, a particular spatial input software may not be compatible witha particular direct input device that a user may want to use.Embodiments disclosed herein may allow a user to use a direct inputdevice despite the spatial software program's incompatibility with thedirect input device.

In an embodiment, the system includes a direct input device (the“client”) executing a client application (“app”), a secondary device(the “server”) executing a server app, and a network link between theserver and the client. The server app transmits to the client app, viathe network link, a portion of the content displayed on the server'scomputer screen. The client app may display this portion. Whenever theuser interacts with the client device, the client app transmits thoseinteractions to the server app over the network link. The server appsends an acknowledgement to the client app and provides the receivedinteractions to the spatial input software executing on the server, andthe spatial input software consumes the interactions as if the userperformed the interactions on the server device itself. Once the serverapp consumes these inputs, the inputs are then reflected in the contentdisplayed on the server's computer screen and thus will be sent to theclient.

The display of the server is typically larger than the display of theclient; sometimes the server's display is many times larger than theclient's display. Scaling the entire contents of the server's display tofit the client's display may produce negative results. First, userinterface elements that are appropriately sized on the server's displaymay be too small when scaled onto the client device for the user tointeract with effectively. Second, a small input movement on the clientdevice scales to a much larger input movement on the server, thusimpeding precise input movements. Embodiments disclosed herein mayinclude a screen selection feature that allows a user to select aportion of the secondary device's display that is to be mirrored ontothe display of the direct input device.

The use of network communications between the server and the clientintroduces several potential problems, such as variable network latency(e.g., the amount of time it takes for a packet of data from one networknode to another), network jitter (e.g., variation in the delay ofreceived data packets), and data packet loss. The client display is acopy of the server display and it may take time to send the input to theserver, process the input on the server, and send back an updateddisplay. If the network delay or server processing delay is significant,the display output on the client device may appear to lag or jump, thusdegrading the user experience and making some tasks, such as drawing,unacceptably difficult.

Network conditions are difficult to predict, even for a simple networkof one server and one client device. To ameliorate the effects of poornetwork conditions on the user experience, a concept of “interim input”may be introduced. Interim input is client device input that the clientapp has sent to the server app and for which the client app has receivedan acknowledgement from the server app, but which may not yet have beenconsumed by the spatial input software executing on the server. Thiscould occur for various reasons, for example the spatial input softwaremay be performing time-consuming processing.

The client application may render the interim input over the renderingof the server's display on the client device at the input location. Theclient application may display interim input distinctly from inputalready consumed by the server. A server that consumes the input updatesthe rendering of the display on the server. This updated rendering isthen propagated to the client, where it will be displayed as rendered bythe server. Distinctly displaying the interim input may be the result ofthe client application not having knowledge of the way the input is tobe rendered at the server (e.g., line color, line thickness, brushpatterns, alpha values, and the like). In other examples, distinctlydisplaying the interim input may be intentional to provide a visual cueto the user that the input is not yet consumed. For example, the regularinput is a solid line, the interim input may be displayed as a dashedline. As another example, if the regular input is a black line, theinterim input may be displayed as a magenta line. The interim input isdisplayed for a sufficient amount of time to hide the network latency;after this time has elapsed, the interim input is no longer rendered bythe client as the input is displayed as regular input (as the server has“caught up” and is now sending the input rendered as part of the datadescribing the server display). This interim input hides the lag thatnormally would occur waiting for the server to consume the input andsend a new display to the client.

In other examples, rather than displaying the interim input distinctly,the client may determine one or more characteristics of the interiminput to match the interim input with nearby previously consumed input.For example, if the user is drawing a line, the client may attempt tomatch the interim input portion of the line with the portion of the linealready consumed by the server. It may do this by readingcharacteristics of pixels corresponding to consumed input that areadjacent (e.g., within a predetermined threshold) to the location of theinterim input in the data describing the server's display (e.g., MPEG orother display frame). These characteristics may be used to match thecharacteristics of nearby consumed input. Characteristics may includecolor, line thickness, brush patterns, alpha values, and the like. Theinterim input may then be rendered using these characteristics.

The amount of time to display the interim input may be a specifiedvalue. Under severely poor network conditions, a concept of“unacknowledged input” may be introduced to further ameliorate theeffects of poor network conditions on the user experience.Unacknowledged input is a portion of the client device input that theclient app has transmitted to the server app, but for which the clientapp has not received an acknowledgement from the server app. In someexamples, the unacknowledged input may be displayed distinctly from theinterim input and displayed distinctly from the regular input. Forexample, if the interim input is displayed as a dashed line, theunacknowledged input may be displayed as a dotted line. As anotherexample, if the interim input is displayed as a magenta line, theunacknowledged input may be displayed as a red line. Upon the client appreceiving from the server app an acknowledgement for a portion of theunacknowledged input, that portion of the unacknowledged input becomesinterim input, and is displayed accordingly. In other examples, theunacknowledged input may be displayed the same as interim input. In someexamples, the unacknowledged input may also be matched to nearbyconsumed input as previously described for interim input.

FIG. 1 illustrates a system 100 for using a direct input device withspatial input software executing on a server device, according to anembodiment. The system 100 includes a client device 102, a server device132, and a network link 120. The client device 102 executes the clientapp and the server device 132 executes the spatial input software andthe server app. The client device 102 may be a tablet computer, asmartphone, a pen display, or any other direct input device. The serverdevice 132 may be a desktop computer, a laptop computer, a computerserver, or any other type of computer that executes the spatial inputsoftware and the server app. The server device 132 may communicate withone or more input devices (not shown), such as a keyboard, a mouse,touchpad, a trackball, etc.

The network link 120 may be a wired link or a wireless link. A wiredlink may be a Universal Serial Bus (USB) link, an Ethernet link, anRS-232 (serial) link, etc. A wireless link may be an Institute forElectrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi) link, aninfrared link, a Bluetooth link, etc. The physical distance separatingthe client device 102 from the server device 132 may range from aslittle as a few inches to as much as thousands of miles. Networkcommunications over network link 120 may be compressed, encrypted, orboth. Furthermore, the network link 120 may be multiple network linkswithin a network, within multiple networks, or even the Internet.

Being a direct input device, the client device 102 includes a clientdisplay 104 that accepts input as well as displays output. The clientdisplay 104 may accept input from a variety of spatial input devices106, such as a pen, a stylus, a finger or thumb, etc. The server device132 includes a server display 135, which may or may not accept inputdirectly. The client display 104 has an aspect ratio, termed a “clientaspect ratio,” which represents the relation between the width of clientdisplay 104 and the height of client display 104. The server display 135has an aspect ratio, termed a “server aspect ratio,” which representsthe relation between the width of server display 135 and the height ofserver display 135. The client aspect ratio and the server aspect ratiomay or may not be equivalent. In an embodiment, if the client aspectratio and the server aspect ratio are different, a letterbox may beadded around the content to be displayed on the client display 104.

A user may select a portion of the server display 135, on which the userwould like to work. This may be useful, for example, if the user wasdrawing in a graphics program and only needed the canvas but not thesurrounding tool palettes to be displayed on the client display 104. Aselected display portion 134 of the server display 135 is displayed onthe client display 104. Content displayed outside of the selecteddisplay portion 134 of the server display 135, such as smiley face 140,is not displayed on the client display 104. In an embodiment, theselected display portion 134 has an aspect ratio fixed to the aspectratio of the client display 104. A user may use the client device 102 orthe server device 132 to modify the selected display portion 134 ofserver display 135, for example by panning or zooming the selecteddisplay portion 134 of server display 135.

To select the selected display portion 134 of the server display 135, aninterface may be provided that appears to sit on top of the display andappears as a graphics tool selection (e.g., the “marching ants”). Usingthis interface, the user is able to drag and resize this selection tochange the selected display portion 134 of the server display 135 thatis to be displayed on the client display 104. The selected displayportion 134 may be changed from the client display 104 in real-time (ornear real-time) by providing the user an interface element on the clientdisplay 104 that switches the mode of the selected display portion 134.If the selected display portion 134 is in the “move” mode, the user mayuse the client display 104 to pan or zoom the selected display portion134, which modifies the selected display portion 134 of the serverdisplay 135.

When a user changes the selected display portion 134 using the clientdisplay 104, the client device 102 sends deltas to the server device132. For example, using the client display 104 to move the selecteddisplay portion 134 up by 100 pixels and across by 150 pixels wouldresult in a delta of (150, 100) being sent by the client device 102 tothe server device 132. The server device 132 receives the deltas andadjusts the selected display portion 134 of the server display 135appropriately. Deltas are also sent to the server device 132 when a userchanges the size of the selected display portion 134 via the clientdisplay 104. In an embodiment, if the selected display portion 134 ismoved outside the bounds of the display, the portion outside the boundsof the display is displayed as black.

After the selected display portion 134 has been selected, the serverdevice 132 begins mirroring the selected display portion 134 to theclient device 102. A video codec is used to send the data across thenetwork link 120. In an embodiment, mirroring is performed by thefollowing process:

1. The server device 132 copies the frame buffer of the server display135.

2. The server device 132 cuts out the selected display portion 134 fromthe frame buffer copy and scales the selected display portion 134 to fitthe client display 104.

3. The server device 132 feeds the new frame, which corresponds to theselected display portion 134, into a video codec (e.g., Moving PicturesExperts Group (MPEG) MPEG-2, 11.264, Motion Joint Photographic ExpertsGroup (Motion JPEG), etc.), which compresses it.

4. The server device 132 divides the compressed output into datagramsfor transmission to the client device 102.

5. The server device 132 transmits the datagrams to the client device102.

6. The client device 102 reassembles the datagrams into one large datablock.

7. The client device 102 feeds the one large data block into a videocodec, which decompresses it.

8. The decompressed output is blitted to the client display 104.

This process is performed repeatedly and quickly (e.g., up to 60 framesper second) to make the client display 104 feel responsive to the user.

The selected display portion 134 not only defines the portion of theserver display 135 to be mirrored onto client display 104, but alsodefines a mapping of interactive events on the client display 104 to theserver device 132. The client device 102 might not be aware of themapping at all. Instead, the client device 102 sends both its screensize and the raw input event information to the server device 132. Theserver device 132 knows the position of the selected display portion 134relative to the server display 135, and scales/maps the received inputevents to fit within the selected display portion 134 of server display135. After scaling/mapping an input event, the spatial input softwareexecuting on server device 132 consumes the scaled/mapped input event asif it was generated by an input device connected to the server device132:

1. On initial connection, the client device 102 sends the size of theclient display 104 to server device 132 across network link 120.

2. A user-initiated input event occurs on the client device 102 (e.g., atouch/pen stroke generating position data).

3. The client device 102 sends the input event across network link 120to server device 132.

4. The server device 132 uses the ratio of the client displaysize/selected display portion 134 to scale the input event (e.g.,client's X1, Y1 scaled to server's X2, Y2).

5. The server device 132 translates the client's X,Y origin so that italigns with the selected display portion 134.

6. The scaled input event is delivered through a system ApplicationProgramming Interface (API) so that spatial input software executing onthe server device 132 consumes the scaled/mapped input event as if itwas generated by an input device connected to the server device 132. Forexample, a system API provided by an operating system.

This process may be performed for each input event generated by clientdevice 102 that contains X,Y position information.

In an embodiment illustrated in FIG. 1B, a navigator 150, which shows aminiature replica of the entire server display 135, may be displayed onthe client display 104. Interactions with (e.g., inputs directed to) thenavigator 150, via client display 104, may move or resize the selecteddisplay portion 134 of the server display 135 and cause a correspondingchange in the selected display portion that is displayed on the clientdisplay 104. The server device 132 sends to the client device 102 asmall thumbnail image of the entire server display 135 and thecoordinates of the selected display portion 134. The client display 104would render the thumbnail and render the selected display portion 134on top of (e.g., in front of) the thumbnail. Any interactions with thenavigator 150 would cause deltas in the X, Y directions to be sent tothe server device 132, as described above.

Returning to FIG. 1, a spatial input device 106, such as a stylus or auser's finger (not shown), may be used by the user to input data intothe client device 102. As the user inputs data into client device 102,the client app transmits the input data to the server device 132 overthe network link 120. Upon receiving the transmitted input data, theserver app sends an acknowledgement to the client device 102 over thenetwork link 120, maps the received input data to server input datacorresponding to the selected display portion 134, and provides theserver input data to the spatial input software executing on the serverdevice 132. The spatial input software consumes the server input data asif the user input the server input data on the server device 132 itself.For example, server device 132 draws line 138 as a result of the userdrawing line 108 on client device 102. Once the server 132 consumes theinput data, the input data may be reflected on the server's display,which is then forwarded to the client for display on the client display.In some examples, prior to this happening, the client app displays theinput 110 on client display 104. As shown in FIG. 2, how the client appdisplays the input may depend on the state of the input on the serverdevice 132. Pointer 136 shows a current position of an input device onthe server device 132.

FIG. 2 illustrates a series of user inputs onto the client device 102,according to an embodiment. The series of user inputs are in the form ofa freeform drawn line, which is displayed as a series of connected linesegments 202, 204, 206. Arrows 210, 212, 214, and 216 show thechronological motions of the spatial input device 106 used to input thefreeform drawn line.

First line segment 202 is displayed as a solid line, denoting that thefirst line segment 202 is “regular input.” The second line segment 204is displayed as a dashed line, denoting that the second line segment 204is “interim input.” The third line segment 206 is displayed as a dottedline, denoting that the third line segment 206 is “unacknowledgedinput.”

The amount of time to display the interim input may be a specifiedvalue, for example 125 ms; after the specified amount of time haselapsed, the interim input is no longer specially rendered by the client(hopefully the server has consumed the input and sent an update of theserver display such that the interim input is displayed as regularinput). The length of the displayed interim input may be proportional tohow much data is input during this specified amount of time; thus, themore data that is input during this specified amount of time, the longerthe length of the displayed interim input line. To provide the user witha visual reminder of the status of a portion of input, the interim inputmay be displayed distinctly from the regular input. For example, in FIG.2, the regular input is first line segment 202 and is displayed as asolid line whereas the interim input is second line segment 204 and isdisplayed as a dashed line.

When the conditions of network link 120 is particularly poor,acknowledgements from the server device 132 for a portion of the inputdata might not be received by the client app before the client device102 renders the input data portion on the client display 104. In suchcircumstances, the portion(s) of the input data are “unacknowledgedinput,” and may be displayed distinctly from both the interim input andthe regular input. For example, in FIG. 2, the unacknowledged input isthird line segment 206 and is displayed as a dotted line. When theclient app receives an acknowledgement from the server device 132 for aportion of the unacknowledged input, that portion of the unacknowledgedinput becomes interim input and is displayed as interim input.

FIG. 3 illustrates how inputs events flow within the system 100,according to an embodiment. The diagram illustrates the client app 302executing on the client device 102, the server app 304 executing onserver device 132, and network communications protocol 320 operatingover network link 120.

When the client app 302 and the server app 304 are started, the two apps302, 304 begin a “pairing” process in which they find each other vianetwork link 120. Upon completing the pairing process, the client app302 and the server app 304 are ready to perform the functions describedbelow. A user session is a period of time starting at the pairing of theclient app 302 and the server app 304 and ending at the disconnectbetween the client app 302 and the server app 304. When the user isfinished using the spatial input software, the user may end the usersession by closing the client app 302 and/or the server app 304.

Input event 306 represents an instance of data input into client device102. The input event 306 may be a drawing input, a keyboard input, agesture input, etc. The event input event 306 is passed to input eventhandler 308. The input event handler 308 may be code that is written fora specific spatial input device 106 (e.g., a specific stylus, any stylusfrom a particular manufacturer, etc.). The input event handler 308contains code for communicating with the spatial input device 106. Theinput event handler 308 passes the input event 306 to event inputmanager 310.

Event input manager 310 dispatches the input event 306 to the UIrenderer 312 and client EventStream 314. The UI renderer 312 isresponsible for rendering input events 306 on the client device 102. Theclient EventStream 314 is responsible for sending input events 306across the network link 120 to the server EventStream 316 executingwithin the server app 304 on server device 132.

Upon receiving an input event 306 from the client EventStream 314, theserver EventStream 316 sends an acknowledgement to the clientEventStream 314 and dispatches the received input event 306 to the eventrendering engine 318. Event rendering engine 318 consumes the receivedinput event 306 and replays it on the server device 132 as if thereceived input event 306 was actually input onto server device 132. Forexample, event rendering engine 318 would convert a drawing input eventreceived by server EventStream. 316 into one or more “mouse down” and“mouse move” events on server device 132.

In some examples, the client EventStream 314 and the server EventStream316 communicate with each other in real-time (or near real-time); thus,the client EventStream 314 and the server EventStream 316 use areal-time (or near real-time) communications protocol over network link120. If network link 120 is an Internet Protocol (IP) based network(whether wired or wireless), the client EventStream 314 and the serverEventStream may send messages (called datagrams) to each other via theUser Datagram Protocol (UDP). Since UDP does not guarantee packet(datagram) delivery, order, or duplicate protection the clientEventStream 314 and the server EventStream 316 may be configured tohandle these situations appropriately. If network link 120 is aBluetooth connection, the client EventStream 314 and the serverEventStream 316 may communicate using the Bluetooth equivalent of UDP.

Event input manager 310 may also maintain an event history. The eventhistory may include a history of input events 306 received from inputevent handler 308, a history of input events 306 sent to the server app304 executing on server device 132, and a history of acknowledgementsreceived from the server app 304 executing on server device 132. Theevent history may be a buffer, an array, a queue, or any other datastructure suitable for its purpose.

When an input event 306 is passed to the event input manager 310, theevent input manager 310 adds the input event 306 to the event history.The event input manager 310 may be informed by the client EventStream314 when the client EventStream 314 receives from the server EventStream316 an acknowledgement that the server EventStream 316 has received aninput event 306. Upon receipt of the acknowledgement, the event inputmanager 310 marks the corresponding input event 306 for removal from itsevent history and starts a timer for that input event 306. The timer isset to a specified amount of time (e.g., 150 ms). When the timer for acorresponding acknowledged input event 306 expires, the event inputmanager 310 removes the corresponding acknowledged input event 306 fromthe event history.

The UI renderer 312 uses the acknowledged input events 306 within theevent history of event input manager 310 to render the interim input. Ifnecessary, the UI renderer 312 uses the unacknowledged input events 306to render the unacknowledged input

FIG. 4 is a portion of a class diagram illustrating relationshipsamongst object-oriented classes for various spatial input devices 106,according to an embodiment. Each class may be responsible forcommunicating with its corresponding hardware and for producingcorresponding events. In some examples, the parent class Pen 402 is thebase class, from which all 44other pens/styli classes inherit. The Pen402 class may include methods and/or variables that are common to allpens/styli.

The FingerPen 404 class is responsible for producing events when theuser produces an input event with the user's finger/thumb. Allmanufactured pens/styli (e.g., not human fingers/thumbs) inherit fromthe SDKPen 406 class, which may include methods and/or variables thatare common to all manufactured pens/styli. Each manufacturer ofpens/styli may have one or more vendor-specific subclasses 408, 410, 412that is specific to that vendor's pens/styli.

When a stylus is paired with the client app 302, the client app 302activates the appropriate subclass so that the subclass can begincommunicating with the stylus hardware and producing input events 306.

Events

An input event 306 is represented by an event structure.

typedef enum {  EventUnknown,  EventStrokeBegin,  EventStrokePoint, EventStrokeEnd,  EventKeyPressed,  ... } EventType;

The EventType enumeration enumerates various event types that may begenerated by the spatial input device 106. An input event 306 with anEventStrokeBegin event type denotes that the input event 306 was thefirst input event 306 of an input stroke on client device 102. An inputevent 306 with an EventStrokeEnd event type denotes that the input event306 was the last input event 306 (e.g., the finger/pen/stylus was liftedoff of the client display 104) of the corresponding input stroke onclient device 102. An input event 306 with an EventStrokePoint eventtype denotes that the input event 306 was neither the first nor the lastinput event 306 of the corresponding pen stroke on client device 102.

typedef struct {  integer identifier;  EventType eventType; }EventHeaderStruct;

Each input event 306 may include an EventHeaderStruct structure, whichmay include the event type and an identifier for the input event 306.The identifier may be an integer, may be signed or unsigned, and may beof an appropriate length (e.g., 32-bit, 64-bit, 128-bit, etc.) touniquely identify an input event 306 within a user session. Within auser session, the identifier is incremented to accommodate the nextinput event 306. In an embodiment, the identifier is incrementedmonotonically.

typedef struct {  float x;  float y;  #ifdef 3D_ENABLED  float z; #endif  float pressure;  #ifdef TILT_ENABLED  float tiltX;  floattiltY;  #endif } EventPointStruct;

Each input event 306 may include an EventPointStruct structure, whichincludes various values for the input event 306. For example, theEventPointStruct structure includes values x and y, which denote the xand y coordinates of the input event 306 on a Cartesian coordinatesystem corresponding to the client display 104. If the input space ofthe client device 102 is three-dimensional, the EventPointStructstructure may include a value for the z coordinate. If either the clientdevice 102 or the spatial input device 106 has a sensor to detect thepressure with which an event was input, the EventPointStruct structuremay include a value for pressure. If either the client device 102 or thespatial input device 106 has a sensor to detect a tilt of the spatialinput device 106 when the spatial input device 106 produced the inputevent 306, the EventPointStruct structure may include a value for tiltXand/or tiltY relative to the client device 102. The various values ofthe EventPointStruct structure may be floating point values.

typedef struct {  EventHeaderStruct header;  EventPointStruct point; }EventStruct;

Each input event 306 may be represented by an EventStruct structure,which includes an EventHeaderStruct structure and an EventPointStructstructure.

FIG. 5 illustrates a client event buffer 510 being populated with inputevents generated by a stylus stroke 502, according to an embodiment. Theclient EventStream 314 and the server EventStream 316 may each includean “event buffer,” 510, 512 which is a data structure for tracking inputevents 306. The event buffer 510, 512 may be a ring buffer, a circularqueue, or any other data structure suitable for its purpose. If an eventbuffer 510, 512 is a ring buffer or a circular queue, the event buffer510, 512 will have two pointers: a tail pointer 504, 514 and a headpointer 506, 516. The pseudocode below gives an example of theEventStream implemented in an object-oriented programming language, suchas C++. Although the tail pointers 504, 514 and the head pointer 506,516 may be implemented as actual pointers, in an embodiment, the tailpointers 504, 514 and the head pointer 506, 516 may be implemented asintegers used to index the event buffer 510, 512.

class EventStream {  int seqStart = 0; //The lowest seq # for the storedevents  int length = 0; // The number of events in eventsBuffer  intcapacity = 1024; // Set to some fixed capacity  EventStructeventsBuffer[capacity];  void addEvent(EventStruct *event) {   int index= (seqStart + length) % capacity;   eventsBuffer[index] = *event;   if(length < (capacity − 1 /* end sentinel */)) {    length++;   }   else {   // stream beyond capacity, dropping old events    seqStart++;   }  } void ackEvent(EventStruct *event) {   int ack =event->header.identifier;   if (ack < seqStart) {    // Ack is stale,nothing to update    return;   }   int maxAck = seqStart + length;   if(ack > maxAck) {    // Ack is from the future? Shouldn't happen, skip   return;   }   length -= ack − seqStart);   seqStart = ack;  } };

Client EventStream 314

When a new input event 306 is produced on the client device 102, theclient EventStream 314 adds the new input event 306 to the head of theclient event buffer 510 (e.g., the space in the client event buffer 510pointed to by head pointer 506). The tail pointer 504 points to the nextinput event 306 that is to be sent to the server EventStream 316. Afterthe client EventStream 314 has sent an input event 306 to the serverEventStream. 316 and has received from the server EventStream 316 anacknowledgement of its receipt, the client EventStream 314 moves thetail pointer 504 forward past the sent input event 306.

In some examples, whenever an input event 306 is added to the clientevent buffer 510, the client EventStream 314 immediately packages upevery unacknowledged input event 306 in the client event buffer 510 andsends all of them to the server EventStream 316. Sending allunacknowledged input events 306 in the client event buffer 510 everytime a new input event 306 is added to the client event buffer 510 maycause some unacknowledged input events 306 to be sent multiple times.Despite this, because the size of each input event 306 is small andbecause the server EventStream 316 knows to how to recognize duplicateinput events 306 having duplicate input events 306 allows the serverEventStream 316 to recover quickly from packet loss.

Server EventStream 316

When the server EventStream 316 receives input events 306, the serverEventStream 316 checks the server event buffer 512 to determine whichinput events 306 have already been “played back” by the event renderingengine 318. This may be done by comparing the identifier of a receivedinput event 306 to the tail pointer 514 and head pointer 516 of theserver event buffer 512. The following pseudocode snippet is an exampleof how the server EventStream 316 may prevent adding duplicate inputevents 306 to server event buffer 512:

// Calculate the largest identifier index in server event buffer RangeselfRange = MakeRange(seqStart, _length): Integer selfMaxIndex =MaxRange(selfRange); // Calculate largest identifier index in thereceived data Range dataRange = MakeRange(dataSequenceIndex,(eventsCount - 1)); Integer dataMaxIndex = MaxRange(dataRange); //seqstart is the start of the identifier nums in the server event bufferif (seqStart >= dataMaxIndex) {  // Nothing to do, no new data to add soreturn early  return; } if (selfMaxIndex >= dataMaxIndex) {  // Nothingto do, no new data to add so return early  return; } // New data hasbeen found! Copy new data into the server event buffer

In the code above, dataSequenceIndex is the number of the startingsequence that was in the most recent packet received by the serverEventStream 316. Similarly segStart is the starting sequence in theserver event buffer.

FIG. 6 illustrates states of event buffers of client EventStream 314 andserver EventStream 316 during event communications, according to anembodiment.

Acknowledging Received Input Events 306

At time T1, the client EventStream. 314 receives input events 306A, 306Band places the input events 306A, 306B into the client event buffer 510.The head pointer 506 is advanced to the next empty slot of the clientevent buffer 510.

At time T2, the client EventStream 314 sends 604 the input events 306A,306B to the server EventStream 316 over the network link 120.

At time T3, upon receiving an input event 306 from the clientEventStream 314, the server EventStream 316 places the input events306A, 306B into the server EventStream 316. The head pointer 516 isadvanced to the next empty slot of the server event buffer 512.

At time T4, the server EventStream 316 sends an acknowledgement 606 tothe client EventStream 314 over the network link 120. Theacknowledgement 606 includes the identifier(s) for the input event(s)306 being acknowledged.

At time T5, the client EventStream 314 receives the acknowledgement 606from the server EventStream 316 and marks for removal the acknowledgedinput events 306A, 306B within the client event buffer 510. The tailpointer 504 is advanced to point to the slot of the client event buffer510 that is just past the most recently acknowledged input event 306.

In an embodiment where input events 306 are assigned monotonicallyincreasing identifiers, the server EventStream 316 may reduce thequantity of acknowledgements it sends to the client EventStream 314 bysending an acknowledgement for the received input event 306 having thehighest identifier rather than sending an acknowledgement for eachreceived input event 306. The tail pointer 504 is advanced to point tothe slot of the client event buffer 510 that is just past theacknowledged input event 306 with the highest identifier.

Out of Order Events

The network communications protocol 320 used by the client EventStream314 and the server EventStream 316 to communicate messages with eachother may be a real-time (or near real-time) communications protocol,which does not guarantee in-order delivery of messages. Thus, an oldinput event 306 may not arrive at the server EventStream 316 until afterthe UI renderer 312 has processed the input event 306. If the serverEventStream 316 were to naively process the old input event 306, theevent rendering engine 318 might create a situation where the old inputevent 306 conflicts with a newer, already processed input event 306(e.g., the input associated with the old input event 306 obscures orcovers the input associated with the newer, already processed inputevent 306.) This problem may be mitigated by the same mechanism that isused to detect duplicate input events 306. When an input event 306 isreceived by the server EventStream 316, the server EventStream 316compares the event identifier for the received input event 306 to theevent identifier for the most recent input event 306 received by theserver EventStream 316. If the event identifier for the received inputevent 306 is lower than the event identifier for the most recent inputevent 306, the server EventStream 316 considers the received input event306 as already processed and drops the received input event 306 as if itwas a duplicate input event 306.

Handling Packet Loss

Many of the input events 306 within client event buffer 510 are sent tothe server EventStream 316 multiple times (e.g., each time an inputevent 306 is added to the client event buffer 510); thus, thisredundancy increases the system's resilience to packet loss. However,packet loss may still occur, for example when the network link(s) 120with poor network conditions (e.g., high latency, high jitter, or highpacket loss).

Packet loss mitigation may be performed as follows. The event inputmanager 310 may keep track of the event identifier for input events 306sent to the server EventStream 316 by client EventStream 314 (in someexamples, all events). In some examples, the event input manager 310only removes (or marks for removal) an input event 306 from the clientevent buffer 510 upon receiving a corresponding acknowledgement from theserver EventStream 316. After a specified time, if the clientEventStream 314 has not received a corresponding acknowledgement fromthe server EventStream 316 for a particular input event 306, the eventinput manager 310 marks the particular input event 306 as lost. When aninput event 306 is marked as lost, in some examples, the clientEventStream 314 resends to the server EventStream 316 all input events306 within the client event buffer 510. This process is repeated untilan acknowledgement is received for the particular input event 306.

UI Renderer 312

The UI renderer 312 renders the user interface on client device 102.Whenever the event input manager 310 receives a new input event 306, theevent input manager 310 provides the UI renderer 312 with the new inputevent 306. The UI renderer 312 uses the coordinates (e.g., x and y ofthe EventPointStruct), the event type, and the acknowledgement status ofthe input event 306 to render the interim input and, if necessary, theunacknowledged input.

FIG. 7 illustrates the event rendering engine 318 dispatching a receivedinput event 306 to one or more render units 702, 704, 706, 712, 722,according to an embodiment. The event rendering engine 318, whichexecutes within the server app 304 on server device 132, is responsiblefor dispatching a received input event 306 to one or more render units.A render unit is an abstract data type that accepts certain types ofreceived input event 306 and simulates the input of a received inputevent 306 on the server device 132. The event rendering engine 318determines, based on the event type of the received input event 306, towhich render unit(s) the received input event 306 should be sent. Anevent type may be associated with more than one render unit; in suchcases, the received input event 306 is sent to each render unitassociated with the event type of the received input event 306.

A render unit registers with the event rendering engine 318 the eventtype(s), for which the render unit should be sent received input events306. For example, the Tablet Render Unit 704 registers with the eventrendering engine 318 the event types EventStrokeBegin, EventStrokePoint,and EventStrokeEnd. Another example, is the Key Render Unit 712registers with the event rendering engine 318 the event typeEventKeyPressed.

Although much of the current disclosure uses example embodiments relatedto drawing, the system is not limited to drawing events. This system maybe used to deliver any kind of user interaction data that must bedelivered to another device in real-time (or near real-time). Forexample, when a user simulates a key press on a client device 102 bypressing a virtual button, the resulting input event 306 is created withan EventKeyPressed event type. Events may also be created for pinchingto zoom on a canvas, panning the canvas, using an eraser on the canvas,etc.

Predictive Input

Some input devices 106 attempt to reduce latency by attempting topredict what the user's next input is going to be. Although these“predicted inputs” cannot be sent to the server app 304 for rendering inthe spatial input software because the “predicted inputs” may beincorrect, the “predicted inputs” could be used to make the interiminput on the client device 102 even more responsive.

When an input event handler 308 receives from input device 106 apredicted input event 306, the input event handler 308 passes thepredicted input event 306 to the event input manager 310. The eventinput manager 310 passes the predicted input event 306 to the UIrenderer 312, which uses the predicted input event 306 to update theinterim input on the client display 104. However, the event inputmanager 310 does not pass the predicted input event 306 to the clientEventStream 314 unless and until the predicted input event 306 isconfirmed to be correct (e.g., by receiving an actual input event 306that sufficiently corresponds to the predicted input event 306).

Appearance of Interim Input and Unacknowledged Input

The interim input or the unacknowledged input may be rendered withvarious colors, sizes, shapes, effects, textures, opacities, etc. Tohide network latency, the client app 302 may match the appearance of theinterim input or the unacknowledged input to the appearance of theregular input (e.g., matching the regular input'scolor/size/shape/effect/texture/opacity/etc.). This would make theinterim input (and possibly the unacknowledged input) indistinguishablefrom the regular input.

Auto Detect Interim Input Time/Length

The amount of time to display the interim input may be a specified.value (e.g., 150 ms). The amount of time to display the interim inputmay be also be dynamically adjusted. For example, the latency of thenetwork link 120 may be automatically detected and used to modifydynamically the amount of time to display the interim input (e.g.,adjusting the amount of time to display the interim input to be directlyproportional to the detected network latency). Dynamicallyadjusting/determining the appropriate interim input time would improvethe user experience of the system and would reduce or eliminate the needfor configuring an appropriate static interim input time.

FIG. 8 is a block diagram illustrating an example of a machine 800, uponwhich any one or more embodiments may be implemented. In alternativeembodiments, the machine 800 may operate as a standalone device or maybe connected (e.g., networked) to other machines. In a networkeddeployment, the machine 800 may operate in the capacity of a servermachine, a client machine, or both in a client-server networkenvironment. In an example, the machine 800 may act as a peer machine ina peer-to-peer (P2P) (or other distributed) network environment. Themachine 800 may implement or include any portion of the systems,devices, or methods illustrated in FIGS. 1-7 and 9-11, and may be apersonal computer (PC), a tablet PC, a set-top box (STB), a personaldigital assistant (PDA), a tablet, a mobile telephone, a smart phone, aweb appliance, a network router, switch or bridge, or any machinecapable of executing instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, although only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein, such as cloud computing, software asa service (SaaS), other computer cluster configurations, etc.

Examples, as described herein, may include, or may operate by, logic ora number of components, modules, or mechanisms. Modules are tangibleentities (e.g., hardware) capable of performing specified operations andmay be configured or arranged in a certain manner. In an example,circuits may be arranged (e.g., internally or with respect to externalentities such as other circuits) in a specified manner as a module. Inan example, the whole or part of one or more computer systems (e.g., astandalone, client or server computer system) or one or more hardwareprocessors may be configured by firmware or software (e.g.,instructions, an application portion, or an application) as a modulethat operates to perform specified operations. In an example, thesoftware may reside on a machine-readable medium. In an example, thesoftware, when executed by the underlying hardware of the module, causesthe hardware to perform the specified operations.

Accordingly, the term “module” is understood to encompass a tangibleentity, be that an entity that is physically constructed, specificallyconfigured (e.g., hardwired), or temporarily (e.g., transitorily)configured (e.g., programmed) to operate in a specified manner or toperform part or all of any operation described herein. Consideringexamples in which modules are temporarily configured, each of themodules need not be instantiated at any one moment in time. For example,where the modules comprise a general-purpose hardware processorconfigured using software, the general-purpose hardware processor may beconfigured as respective different modules at different times. Softwaremay accordingly configure a hardware processor, for example, toconstitute a particular module at one instance of time and to constitutea different module at a different instance of time.

Machine (e.g., computer system) 800 may include a hardware processor 802(e.g., a central processing unit (CPU), a graphics processing unit(GPU), a hardware processor core, or any combination thereof), a mainmemory 804 and a static memory 806, some or all of which may communicatewith each other via an interlink (e.g., bus) 808. The machine 800 mayfurther include a display unit 810, an alphanumeric input device 812(e.g., a keyboard), and a user interface (UI) navigation device 814(e.g., a mouse, a pen/stylus, etc.). In an example, the display unit810, input device 812 and UI navigation device 814 may be a touch screendisplay. The machine 800 may additionally include a storage device(e.g., drive unit) 816, a signal generation device 818 (e.g., aspeaker), a network interface device 820, and one or more sensors 821,such as a global positioning system (GPS) sensor, compass,accelerometer, or other sensor. The machine 800 may include an outputcontroller 828, such as a serial (e.g., universal serial bus (USB),parallel, or other wired or wireless (e.g., infrared (IR), near fieldcommunication (NFC), etc.) connection to communicate or control one ormore peripheral devices (e.g., a printer, card reader, etc.)

The storage device 816 may include a machine-readable medium 822 onwhich is stored one or more sets of data structures or instructions 824(e.g., software) embodying or utilized by any one or more of thetechniques or functions described herein. The instructions 824 may alsoreside, completely or at least partially, within the main memory 804,within static memory 806, or within the hardware processor 802 duringexecution thereof by the machine 800. In an example, one or anycombination of the hardware processor 802, the main memory 804, thestatic memory 806, or the storage device 816 may constitutemachine-readable media.

Although the machine-readable medium 822 is illustrated as a singlemedium, the term “machine-readable medium” may include a singly, mediumor multiple media (e.g., a centralized or distributed database, and/orassociated caches and servers) configured to store the one or moreinstructions 824.

The term “machine-readable medium” may include any medium that iscapable of storing, encoding, or carrying instructions for execution bythe machine 800 and that cause the machine 800 to perform any one ormore of the techniques of the present disclosure, or that is capable ofstoring, encoding or carrying data structures used by or associated withsuch instructions. Non-limiting machine-readable medium examples mayinclude solid-state memories, and optical and magnetic media.Accordingly, machine-readable media are not transitory propagatingsignals. Specific examples of machine-readable media may includenon-volatile memory, such as semiconductor memory devices (e.g.,Electrically Programmable Read-Only Memory (EPROM), ElectricallyErasable Programmable Read-Only Memory (EEPROM)) and flash memorydevices; magnetic disks, such as internal hard disks and removabledisks; magneto-optical disks; Random Access Memory (RAM); Solid StateDrives (SSD); and CD-ROM and DVD-ROM disks.

The instructions 824 may further be transmitted or received over acommunications network 826 using a transmission medium via the networkinterface device 820 utilizing any one of a number of transfer protocols(e.g., frame relay, internet protocol (IP), transmission controlprotocol (TCP), user datagram protocol (UDP), hypertext transferprotocol (HTTP), etc.). The term “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding orcarrying instructions for execution by the machine 800, and includesdigital or analog communications signals or other intangible medium tofacilitate communication of such software.

Turning now to FIG. 9, an example method 900 of a client devicerendering input is shown according to some examples of the presentdisclosure. At operation 902, the client device receives data describinga portion of the server display. For example, the client devicesreceives a frame sent by the server as one or more packets encoded by avideo codec. As noted, this communication may occur over a network. Thisframe may describe a portion of the server's display that is a subset ofthe complete display. For example, server display 135 of FIG. 1, theportion sent may be the portion denoted by the dotted rectangle displayportion 134 in FIG. 1.

At operation 904 the received portion of the server display may berendered on the display of the client. Operations 902 and 904 may beperformed repeatedly to obtain a video frame rate that is smooth andstutter free. For example, 30, 60, or other frames per second. Atoperation 906 an input may be received via an input device (such as atouchscreen from a touch screen display) on the client. The input mayinclude one or more data points that describe one or more properties ofthe input (such as coordinates of the input). At operation 908 datadescribing the input is sent to the server device. In some examples, atthis point, the input may be rendered on the display of the clientdevice in a first manner. At operation 910 the client device receives anacknowledgement from the server device. At operation 912 the input maybe rendered. In some examples, the input may be rendered in a mannerthat distinguishes it from input that is consumed by the server. If theinput was already rendered in a first manner prior to beingacknowledged, the input may be re-rendered in a second manner distinctfrom the first manner. Once the input is considered consumed by theserver (in some examples after a determined waiting period expires) theinput may cease to be rendered by the client application (as the serverapplication would have consumed the input and rendered it there, whichwould reflect on the client).

Turning now to FIG. 10, a method 1000 of changing the portion of theserver display that is displayed on the client device is shown accordingto some examples of the present disclosure. At operation 1002, theclient device may receive an input from the user that the user wishes tochange the portion of the server display displayed by the client. Thisinput may be directed to a graphical user interface (GUI) element on theclient device, such as shown in FIG. 1B. For example, the navigator 150of FIG. 1 B may be overlaid over the currently shown display of theserver. At operation 1004 a command to change the portion of the serverdisplay may be sent to the server. This command may include dataidentifying the new display coordinates or data identifying changes inthe displayed portions (e.g., delta-x and delta-y coordinates). Atoperation 1006, the client device may receive data describing the secondportion of the server display. The first and second portions may besmaller portions than a complete portion of the server display. Atoperation 1008, the second portion may be rendered to the display screenof the client device.

Turning now to FIG. 11, a method 1100 of a server processing an inputevent sent from a client device is shown according to some examples. Atoperation 1102 the server sends the client a portion of the serverdisplay. This portion may be selected by the client on the clientdevice. The server may encode and packetize the portion of the displaybuffer of interest and send it to the client device. In some examples,this may be done continuously, or quasi-continuously to ensure a highframe rate (e.g., 30, 60 fps). At operation 1104 a client event isreceived. Data describing the client event, such as coordinates, may beincluded in the event. At operation 1106, the coordinates in the eventsent by the client may be converted to coordinates of the serverdisplay. This may be done as previously described by utilizing the ratioof the client display size to the selected display portion. At operation1108 the client event may be consumed for example, as a local inputwould be consumed. This may include rendering the event. At operation1110 an updated display may be sent to the client. As noted previously,the communications between the server and client may take place over oneor more networks.

Additional Notes & Example Embodiments

Conventional terms in the fields of computer networking and computersystems have been used herein. The terms are known in the art and areprovided only as a non-limiting example for convenience purposes.Accordingly, the interpretation of the corresponding terms in theclaims, unless stated otherwise, is not limited to any particulardefinition.

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat any arrangement that is calculated to achieve the same purpose maybe substituted for the specific embodiments shown. Many adaptations willbe apparent to those of ordinary skill in the art. Accordingly, thisapplication is intended to cover any adaptations or variations.

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration, specific embodiments that may bepracticed. These embodiments are also referred to herein as “examples.”Such examples may include elements in addition to those shown ordescribed. However, the present inventors also contemplate examples inwhich only those elements shown or described are provided. Moreover, thepresent inventors also contemplate examples using any combination orpermutation of those elements shown or described (or one or more aspectsthereof), either with respect to a particular example (or one or moreaspects thereof), or with respect to other examples one or more aspectsthereof) shown or described herein.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B,” unlessotherwise indicated. Moreover, in the following claims, the terms“first,” “second,” and “third,” etc. are used merely as labels, and arenot intended to impose numerical requirements on their objects.

In this Detailed Description, various features may have been groupedtogether to streamline the disclosure. This should not be interpreted asintending that an unclaimed disclosed feature is essential to any claim.Rather, inventive subject matter may lie in less than all features of aparticular disclosed embodiment.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments may be used, such as by one of ordinary skill in the artupon reviewing the above description.

Other notes and examples

Example 1 is a computing device comprising: a display screen; an inputdevice; a processor; a memory coupled to the processor and comprisinginstructions, when performed by the processor, causes the computingdevice to perform operations comprising: receiving from a server device,data describing a portion of a graphical display of the server device;rendering the portion of the graphical display of the server device onthe display screen using the received data; receiving an input via theinput device; sending data describing the input to the server device,the server device processing the input from the input device of thecomputing device as local input on the server device; receiving anacknowledgement from the server device that the data describing theinput was received; and responsive to the acknowledgement from theserver device, rendering the input by the processor over the portion ofthe graphical display of the server device displayed on the displayscreen at a position of the input.

In Example 2, the subject matter of Example 1 optionally includeswherein the operations comprise: ceasing to render the input over theportion of the graphical display of the server displayed on the displayscreen when a determination is made that the server device has consumedthe input.

In Example 3, the subject matter of Example 2 optionally includeswherein the operations of ceasing to render the input over the portionof the graphical display of the server displayed on the display screenwhen a determination is made that the server device has consumed theinput comprises: waiting a predetermined period of time; and uponconclusion of the predetermined period of time, determining that theserver device has consumed the input; and in response, ceasing to renderthe input over the portion of the graphical display of the serverdisplayed on the display screen.

In Example 4, the subject matter of Example 3 optionally includeswherein the operations comprise: receiving from the server device,second data describing the portion of the graphical display on theserver device; and rendering the portion of the graphical display of theserver device on the display screen using the received second data, therendered portion of the graphical display of the server device includinga visual representation of the input that is visible after ceasing torender the input over the portion of the graphical display.

In Example 5, the subject matter of any one or more of Examples 1-4optionally include wherein the operations comprise: determiningcharacteristics of the input by determining how previously consumedinputs near the position of the input were displayed by examining thereceived data describing the portion of the graphical display of theserver device; and wherein rendering the input by the processor over theportion of the graphical display of the server device displayed on thedisplay screen at the position of the input comprises rendering theinput using the determined characteristics.

In Example 6, the subject matter of any one or more of Examples 1-5optionally include wherein the operations comprise: prior to a receiptof an acknowledgement from the server device that the data describingthe input was received, rendering, in a first manner, the input by theprocessor over the portion of the graphical display of the server devicedisplayed on the display screen at the position of the input; receivingan acknowledgement from the server device that the data describing theinput was received; responsive to receipt of the acknowledgement fromthe server device, re-rendering in a second manner, the input by theprocessor over the portion of the graphical display of the server devicedisplayed on the display screen at the position of the input; andresponsive to a determination that the server device has consumed theinput, ceasing to render the input over the portion of the graphicaldisplay, the first and second manners being visually distinct.

In Example 7, the subject matter of any one or more of Examples 1-6optionally include wherein the operations comprise: receiving an inputvia the input device to change the portion of the graphical displayreceived from the server device to a second portion; receiving from theserver device second data describing the second portion of the graphicaldisplay of the server device, the second portion being different thanthe portion; and rendering the second portion of the graphical displayof the server device on the display screen using the second receiveddata.

In Example 8, the subject matter of Example 7 optionally includeswherein the input to change the portion of the graphical displayreceived from the server device to the second portion comprises an inputdirected to a navigation display selecting the second portion.

In Example 9, the subject matter of Example 8 optionally includeswherein the operations comprise: sending a command to the server deviceto change to the second portion, the command comprises a delta x and adelta y value describing a difference between the portion and the secondportion.

In Example 10, the subject matter of any one or more of Examples 8-9optionally include wherein the navigation display is overlaid on top ofthe rendered received portion of the graphical display of the serverdevice on the display screen.

Example 11 is a method comprising: using a processor: receiving from aserver device, data describing a portion of a graphical display of theserver device; rendering the portion of the graphical display of theserver device on a display screen using the received data; receiving aninput via an input device; sending data describing the input to theserver device, the server device processing the input from the inputdevice as local input on the server device; receiving an acknowledgementfrom the server device that the data describing the input was received;and responsive to the acknowledgement from the server device, renderingthe input by the processor over the portion of the graphical display ofthe server device displayed on the display screen at a position of theinput.

In Example 12, the subject matter of Example 11 optionally includesceasing to render the input over the portion of the graphical display ofthe server displayed on the display screen when a determination is madethat the server device has consumed the input.

In Example 13, the subject matter of Example 12 optionally includeswherein ceasing to render the input over the portion of the graphicaldisplay of the server displayed on the display screen when adetermination is made that the server device has consumed the inputcomprises: waiting a predetermined period of time; and upon conclusionof the predetermined period of time, determining that the server devicehas consumed the input; and in response, ceasing to render the inputover the portion of the graphical display of the server displayed on thedisplay screen.

In Example 14, the subject matter of Example 13 optionally includesreceiving from the server device, second data describing the portion ofthe graphical display on the server device; and rendering the portion ofthe graphical display of the server device on the display screen usingthe received second data, the rendered portion of the graphical displayof the server device including a visual representation of the input thatis visible after ceasing to render the input over the portion of thegraphical display.

In Example 15, the subject matter of any one or more of Examples 11-14optionally include determining characteristics of the input bydetermining how previously consumed inputs near the position of theinput were displayed by examining the received data describing theportion of the graphical display of the server device; and whereinrendering the input by the processor over the portion of the graphicaldisplay of the server device displayed on the display screen at theposition of the input comprises rendering the input using the determinedcharacteristics,

In Example 16, the subject matter of any one or c ore of Examples 11-15optionally include prior to a receipt of an acknowledgement from theserver device that the data describing the input was received,rendering, in a first manner, the input by the processor over theportion of the graphical display of the server device displayed on thedisplay screen at the position of the input; receiving anacknowledgement from the server device that the data describing theinput was received; responsive to receipt of the acknowledgement fromthe server device, re-rendering in a second manner, the input by theprocessor over the portion of the graphical display of the server devicedisplayed on the display screen at the position of the input; andresponsive to a determination that the server device has consumed theinput, ceasing to render the input over the portion of the graphicaldisplay, the first and second manners being visually distinct.

In Example 17, the subject matter of any one or more of Examples 11-16optionally include wherein the method further comprises: receiving aninput via the input device to change the portion of the graphicaldisplay received from the server device to a second portion; receivingfrom the server device second data describing the second portion of thegraphical display of the server device, the second portion beingdifferent than the portion; and rendering the second portion of thegraphical display of the server device on the display screen using thesecond received data.

In Example 18, the subject matter of Example 17 optionally includeswherein the input to change the portion of the graphical displayreceived from the server device to the second portion comprises an inputdirected to a navigation display selecting the second portion.

In Example 19, the subject matter of Example 18 optionally includeswherein the method further comprises: sending a command to the serverdevice to change to the second portion, the command comprises a delta xand a delta y value describing a difference between the portion and thesecond portion.

In Example 20, the subject matter of any one or more of Examples 18-19optionally include wherein the navigation display is overlaid on top ofthe rendered received portion of the graphical display of the serverdevice on the display screen.

Example 21 is a machine-readable medium, comprising instructions, whichwhen performed by a machine, causes the machine to perform operationscomprising: receiving from a server device, data describing a portion ofa graphical display of the server device; rendering the portion of thegraphical display of the server device on a display screen using thereceived data; receiving an input via an input device; sending datadescribing the input to the server device, the server device processingthe input from the input device of the computing device as local inputon the server device; receiving an acknowledgement from the serverdevice that the data describing the input was received; and responsiveto the acknowledgement from the server device, rendering the input bythe processor over the portion of the graphical display of the serverdevice displayed on the display screen at a position of the input.

In Example 22 the subject matter of Example 21 optionally includeswherein the operations comprise: ceasing to render the input over theportion of the graphical display of the server displayed on the displayscreen when a determination is made that the server device has consumedthe input.

In Example 23, the subject matter of Example 22 optionally includeswherein the operations of ceasing to render the input over the portionof the graphical display of the server displayed on the display screenwhen a determination is made that the server device has consumed theinput comprises: waiting a predetermined period of time; and uponconclusion of the predetermined period of time, determining that theserver device has consumed the input; and in response, ceasing to renderthe input over the portion of the graphical display of the serverdisplayed on the display screen.

In Example 24, the subject matter of Example 23 optionally includeswherein the operations comprise: receiving from the server device,second data describing the portion of the graphical display on theserver device; and rendering the portion of the graphical display of theserver device on the display screen using the received second data, therendered portion of the graphical display of the server device includinga visual representation of the input that is visible after ceasing torender the input over the portion of the graphical display.

In Example 25, the subject matter of any one or more of Examples 21-24optionally include wherein the operations comprise: determiningcharacteristics of the input by determining how previously consumedinputs near the position of the input were displayed by examining thereceived data describing the portion of the graphical display of theserver device; and wherein rendering the input by the processor over theportion of the graphical display of the server device displayed on thedisplay screen at the position of the input comprises rendering theinput using the determined characteristics.

In Example 26, the subject matter of any one or more of Examples 21-25optionally include wherein the operations comprise: prior to a receiptof an acknowledgement from the server device that the data describingthe input was received, rendering, in a first manner, the input by theprocessor over the portion of the graphical display of the server devicedisplayed on the display screen at the position of the input; receivingan acknowledgement from the server device that the data describing theinput was received; responsive to receipt of the acknowledgement fromthe server device, re-rendering in a second manner, the input by theprocessor over the portion of the graphical display of the server devicedisplayed on the display screen at the position of the input; andresponsive to a determination that the server device has consumed theinput, ceasing to render the input over the portion of the graphicaldisplay, the first and second manners being visually distinct.

In Example 27, the subject matter of any one or more of Examples 21-26optionally include wherein the operations comprise: receiving an inputvia the input device to change the portion of the graphical displayreceived from the server device to a second portion; receiving from theserver device second data describing the second portion of the graphicaldisplay of the server device, the second portion being different thanthe portion; and rendering the second portion of the graphical displayof the server device on the display screen using the second receiveddata.

In Example 28, the subject matter of Example 27 optionally includeswherein the input to change the portion of the graphical displayreceived from the server device to the second portion comprises an inputdirected to a navigation display selecting the second portion.

In Example 29, the subject matter of Example 28 optionally includeswherein the operations comprise: sending a command to the server deviceto change to the second portion, the command comprises a delta x and adelta y value describing a difference between the portion and the secondportion.

In Example 30, the subject matter of any one or more of Examples 28-29optionally include wherein the navigation display is overlaid on top ofthe rendered received portion of the graphical display of the serverdevice on the display screen.

Example 31 is a computing device comprising: a display screen; an inputdevice; a processor; a memory coupled to the processor and comprisinginstructions, when performed by the processor, causes the computingdevice to perform operations comprising: receiving from a server device,data describing a first portion of a graphical display on the serverdevice, the first portion being less than a complete graphical displayon the server device; rendering the first portion of the graphicaldisplay of the server device on the display screen using the receiveddata describing the first portion; receiving an input via the inputdevice; sending data describing the input to the server device, theserver device processing the input from the input device of thecomputing device as local input on the server device; receiving an inputvia the input device from a graphical user interface (GUI) elementdisplayed on the display screen to change the first portion of thegraphical display received from the server device to a second portion;transmitting a command to the server device to change the first portionof the graphical display received from the server device to the secondportion; receiving from the server device second data describing thesecond portion of the graphical display of the server device, the secondportion being less than the complete graphical display on the serverdevice and being different than the first portion; and rendering thesecond portion of the graphical display of the server device on thedisplay screen using the received second data.

In Example 32, the subject matter of Example 31 optionally includeswherein the operations further comprise: receiving an acknowledgementfrom the server device that the data describing the input was received;and responsive to the acknowledgement from the server device, renderingthe input by the processor over the first portion of the graphicaldisplay of the server device displayed on the display screen at aposition of the input.

In Example 33, the subject matter of Example 32 optionally includeswherein the operations comprise: ceasing to render the input over thefirst portion of the graphical display of the server displayed on thedisplay screen when a determination is made that the server device hasconsumed the input.

In Example 34, the subject matter of Example 33 optionally includeswherein the operations of ceasing to render the input over the firstportion of the graphical display of the server displayed on the displayscreen when a determination is made that the server device has consumedthe input comprises: waiting a predetermined period of time; and uponconclusion of the predetermined period of time, determining that theserver device has consumed the input; and in response, ceasing to renderthe input over the first portion of the graphical display of the serverdisplayed on the display screen.

In Example 35, the subject matter of Example 34 optionally includeswherein the operations comprise: receiving from the server device,second data describing the first portion of the graphical display on theserver device; and rendering the first portion of the graphical displayof the server device on the display screen using the received seconddata, the rendered first portion of the graphical display of the serverdevice including a visual representation of the input that is visibleafter ceasing to render the input over the first portion of thegraphical display.

In Example 36, the subject matter of any one or more of Examples 31-35optionally include wherein the operations comprise: determiningcharacteristics of the input by determining how previously consumedinputs near the coordinates of the input were displayed by examining thereceived data describing the first portion of the graphical display ofthe server device; and wherein rendering the input by the processor overthe first portion of the graphical display of the server devicedisplayed on the display screen at the position of the input comprisesrendering the input using the determined characteristics.

in Example 37, the subject matter of any one or more of Examples 31-36optionally include wherein the operations comprise: prior to a receiptof an acknowledgement from the server device that the data describingthe input was received, rendering, in a first manner, the input by theprocessor over the first portion of the graphical display of the serverdevice displayed on the display screen at the position of the input;receiving an acknowledgement from the server device that the datadescribing the input was received; responsive to receipt of theacknowledgement from the server device, re-rendering in a second manner,the input by the processor over the first portion of the graphicaldisplay of the server device displayed on the display screen at theposition of the input; and responsive to a determination that the serverdevice has consumed the input, ceasing to render the input over thefirst portion of the graphical display, the first and second mannersbeing visually distinct.

In Example 38, the subject matter of any one or more of Examples 31-37optionally include wherein the operations comprise: wherein the GUIelement comprises a navigation display showing a smaller version of thecomplete graphical display on the server device and an indication of thefirst portion of the graphical display on the server device in relationto the complete graphical display on the server device; and wherein theinput to change the first portion of the graphical display received fromthe server device to the second portion comprises an input directed tothe navigation display selecting the second portion.

In Example 39, the subject matter of Example 38 optionally includeswherein the command transmitted to the server device comprises a delta xand a delta y value describing a difference between the portion and thesecond portion.

In Example 40, the subject matter of any one or more of Examples 38-39optionally include wherein the navigation display is overlaid on top ofthe rendered received portion of the graphical display of the serverdevice on the display screen.

Example 41 is a method comprising: receiving from a server device, datadescribing a first portion of a graphical display on the server device,the first portion being less than a complete graphical display on theserver device; rendering the first portion of the graphical display ofthe server device on a display screen using the received data describingthe first portion; receiving an input via an input device; sending datadescribing the input to the server device, the server device processingthe input from the input device of the computing device as local inputon the server device; receiving an input via the input device from agraphical user interface (GUI) element displayed on the display screento change the first portion of the graphical display received from theserver device to a second portion; transmitting a command to the serverdevice to change the first portion of the graphical display receivedfrom the server device to the second portion; receiving from the serverdevice second data describing the second portion of the graphicaldisplay of the server device, the second portion being less than thecomplete graphical display on the server device and being different thanthe first portion; and rendering the second portion of the graphicaldisplay of the server device on the display screen using the receivedsecond data.

In Example 42, the subject matter of Example 41 optionally includesreceiving an acknowledgement from the server device that the datadescribing the input was received; and responsive to the acknowledgementfrom the server device, rendering the input by the processor over thefirst portion of the graphical display of the server device displayed onthe display screen at a position of the input.

In Example 43, the subject matter of Example 42 optionally includesceasing to render the input over the first portion of the graphicaldisplay of the server displayed on the display screen when adetermination is made that the server device has consumed the input.

In Example 44, the subject matter of Example 43 optionally includeswherein ceasing to render the input over the first portion of thegraphical display of the server displayed on the display screen when adetermination is made that the server device has consumed the inputcomprises: waiting a predetermined period of time; and upon conclusionof the predetermined period of time, determining that the server devicehas consumed the input; and in response, ceasing to render the inputover the first portion of the graphical display of the server displayedon the display screen.

In Example 45, the subject matter of Example 44 optionally includesreceiving from the server device, second data describing the firstportion of the graphical display on the server device; and rendering thefirst portion of the graphical display of the server device on thedisplay screen using the received second data, the rendered firstportion of the graphical display of the server device including a visualrepresentation of the input that is visible after ceasing to render theinput over the first portion of the graphical display.

In Example 46, the subject matter of any one or more of Examples 41-45optionally include determining characteristics of the input bydetermining how previously consumed inputs near the coordinates of theinput were displayed by examining the received data describing the firstportion of the graphical display of the server device; and whereinrendering the input by the processor over the first portion of thegraphical display of the server device displayed on the display screenat the position of the input comprises rendering the input using thedetermined characteristics,

In Example 47, the subject matter of any one or more of Examples 41-46optionally include prior to a receipt of an acknowledgement from theserver device that the data describing the input was received,rendering, in a first manner, the input by the processor over the firstportion of the graphical display of the server device displayed on thedisplay screen at the position of the input; receiving anacknowledgement from the server device that the data describing theinput was received; responsive to receipt of the acknowledgement fromthe server device, re-rendering in a second manner, the input by theprocessor over the first portion of the graphical display of the serverdevice displayed on the display screen at the position of the input; andresponsive to a determination that the server device has consumed theinput, ceasing to render the input over the first portion of thegraphical display, the first and second manners being visually distinct.

In Example 48, the subject matter of any one or more of Examples 41-47optionally include wherein the GUI element comprises a navigationdisplay showing a smaller version of the complete graphical display onthe server device and an indication of the first portion of thegraphical display on the server device in relation to the completegraphical display on the server device; and wherein the input to changethe first portion of the graphical display received from the serverdevice to the second portion comprises an input directed to thenavigation display selecting the second portion.

In Example 49, the subject matter of Example 48 optionally includeswherein the command transmitted to the server device comprises a delta xand a delta y value describing a difference between the portion and thesecond portion.

In Example 50, the subject matter of any one or more of Examples 48-49optionally include wherein the navigation display is overlaid on top ofthe rendered received portion of the graphical display of the serverdevice on the display screen.

Example 51 is a machine-readable medium, comprising instructions, whichwhen performed by a machine, causes the machine to perform operationscomprising: receiving from a server device, data describing a firstportion of a graphical display on the server device, the first portionbeing less than a complete graphical display on the server device;rendering the first portion of the graphical display of the serverdevice on a display screen using the received data describing the firstportion; receiving an input via an input device; sending data describingthe input to the server device, the server device processing the inputfrom the input device of the computing device as local input on theserver device; receiving an input via the input device from a graphicaluser interface (GUI) element displayed on the display screen to changethe first portion of the graphical display received from the serverdevice to a second portion; transmitting a command to the server deviceto change the first portion of the graphical display received from theserver device to the second portion; receiving from the server devicesecond data describing the second portion of the graphical display ofthe server device, the second portion being less than the completegraphical display on the server device and being different than thefirst portion; and rendering the second portion of the graphical displayof the server device on the display screen using the received seconddata.

In Example 52, the subject matter of Example 51 optionally includeswherein the operations further comprise: receiving an acknowledgementfrom the server device that the data describing the input was received;and responsive to the acknowledgement from the server device, renderingthe input by the processor over the first portion of the graphicaldisplay of the server device displayed on the display screen at aposition of the input.

In Example 53, the subject matter of Example 52 optionally includeswherein the operations comprise: ceasing to render the input over thefirst portion of the graphical display of the server displayed on thedisplay screen when a determination is made that the server device hasconsumed the input.

In Example 54, the subject matter of Example 53 optionally includeswherein the operations of ceasing to render the input over the firstportion of the graphical display of the server displayed on the displayscreen when a determination is made that the server device has consumedthe input comprises: waiting a predetermined period of time; and uponconclusion of the predetermined period of time, determining that theserver device has consumed the input; and in response, ceasing to renderthe input over the first portion of the graphical display of the serverdisplayed on the display screen.

In Example 55, the subject matter of Example 54 optionally includeswherein the operations comprise: receiving from the server device,second data describing the first portion of the graphical display on theserver device; and rendering the first portion of the graphical displayof the server device on the display screen using the received seconddata, the rendered first portion of the graphical display of the serverdevice including a visual representation of the input that is visibleafter ceasing to render the input over the first portion of thegraphical display.

in Example 56, the subject matter of any one or more of Examples 51-55optionally include wherein the operations comprise: determiningcharacteristics of the input by determining how previously consumedinputs near the coordinates of the input were displayed by examining thereceived data describing the first portion of the graphical display ofthe server device; and wherein rendering the input by the processor overthe first portion of the graphical display of the server devicedisplayed on the display screen at the position of the input comprisesrendering the input using the determined characteristics.

In Example 57, the subject matter of any one or more of Examples 51-56optionally include wherein the operations comprise: prior to a receiptof an acknowledgement from the server device that the data describingthe input was received, rendering, in a first manner, the input by theprocessor over the first portion of the graphical display of the serverdevice displayed on the display screen at the position of the input;receiving an acknowledgement from the server device that the datadescribing the input was received; responsive to receipt of theacknowledgement from the server device, re-rendering in a second manner,the input by the processor over the first portion of the graphicaldisplay of the server device displayed on the display screen at theposition of the input; and responsive to a determination that the serverdevice has consumed the input, ceasing to render the input over thefirst portion of the graphical display, the first and second mannersbeing visually distinct.

In Example 58, the subject matter of any one or more of Examples 51-57optionally include wherein the operations comprise: wherein the GUIelement comprises a navigation display showing a smaller version of thecomplete graphical display on the server device and an indication of thefirst portion of the graphical display on the server device in relationto the complete graphical display on the server device; and wherein theinput to change the first portion of the graphical display received fromthe server device to the second portion comprises an input directed tothe navigation display selecting the second portion.

In Example 59, the subject matter of Example 58 optionally includeswherein the command transmitted to the server device comprises a delta xand a delta y value describing a difference between the portion and thesecond portion.

In Example 60, the subject matter of any one or more of Examples 58-59optionally include wherein the navigation display is overlaid on top ofthe rendered received portion of the graphical display of the serverdevice on the display screen.

Example 61 is a computing device comprising: a display screen; aprocessor; a memory coupled to the processor and including instructions,when performed by the processor, causes the computing device to performoperations comprising: sending data describing contents of a definedportion of a graphical display of the display screen to a client deviceover a network; receiving information on a user-initiated input eventfrom the client device; modifying client space coordinates in thereceived information to server space coordinates based upon a size ofthe defined portion; consuming the modified information on theuser-initiated input event as if it were an input generated locally;updating the display screen based upon the modified information; andsending data describing contents of the defined portion of the updatedgraphical display of the display screen to the client device over anetwork.

In Example 62, the subject matter of Example 61 optionally includeswherein the operations further comprise: sending an acknowledgement tothe client responsive to receiving the information on a user-initiatedinput event from the client device.

In Example 63, the subject matter of any one or more of Examples 61-62optionally include wherein the operations further comprise: encoding thedata describing contents of the defined portion of the graphical displayof the display screen to the client device over the network using avideo codec.

In Example 64, the subject matter of any one or more of Examples 61-63optionally include wherein the operations further comprise: receiving achange in the defined portion of the graphical display of the displayscreen to a second defined portion; and sending data describing contentsof the second defined portion of the graphical display of the displayscreen to the client device over the network.

In Example 65, the subject matter of any one or more of Examples 61-64optionally include wherein the operations further comprise: storing thereceived information on the user-initiated input event in an eventqueue; and prior to consuming the modified information on theuser-initiated input event, determining that a largest sequence numberof events in the event queue is less than a sequence in the informationon the user-initiated input event.

In Example 66, the subject matter of any one or more of Examples 61-65optionally include wherein the operations further comprise: sending athumbnail representation of the entire graphical display of the displayscreen to the client device over the network.

Example 67 is a method comprising: sending data describing contents of adefined portion of a graphical display of a display screen to a clientdevice over a network; receiving information on a user-initiated inputevent from the client device; modifying client space coordinates in thereceived information to server space coordinates based upon a size ofthe defined portion; consuming the modified information on theuser-initiated input event as if it were an input generated locally;updating the display screen based upon the modified information; andsending data describing contents of the defined portion of the updatedgraphical display of the display screen to the client device over anetwork.

In Example 68, the subject matter of Example 67 optionally includeswherein the operations further comprise: sending an acknowledgement tothe client responsive to receiving the information on a user-initiatedinput event from the client device.

In Example 69, the subject matter of any one or more of Examples 67-68optionally include wherein the operations further comprise: encoding thedata describing contents of the defined portion of the graphical displayof the display screen to the client device over the network using avideo codec.

In Example 70, the subject matter of any one or more of Examples 67-69optionally include wherein the operations further comprise: receiving achange in the defined portion of the graphical display of the displayscreen to a second defined portion; and sending data describing contentsof the second defined portion of the graphical display of the displayscreen to the client device over the network.

In Example 71, the subject matter of any one or more of Examples 67-70optionally include wherein the operations further comprise: storing thereceived information on the user-initiated input event in an eventqueue; and prior to consuming the modified information on theuser-initiated input event, determining that a largest sequence numberof events in the event queue is less than a sequence in the informationon the user-initiated input event.

In Example 72, the subject matter of any one or more of Examples 67-71optionally include wherein the operations further comprise: sending athumbnail representation of the entire graphical display of the displayscreen to the client device over the network.

Example 73 is a machine-readable medium comprising instructions, whichwhen performed by a machine, cause the machine to perform operationscomprising: sending data describing contents of a defined portion of agraphical display of a display screen to a client device over a network;receiving information on a user-initiated input event from the clientdevice; modifying client space coordinates in the received informationto server space coordinates based upon a size of the defined portion;consuming the modified information on the user-initiated input event asif it were an input generated locally; updating the display screen basedupon the modified information; and sending data describing contents ofthe defined portion of the updated graphical display of the displayscreen to the client device over a network.

In Example 74, the subject matter of Example 73 optionally includeswherein the operations further comprise: sending an acknowledgement tothe client responsive to receiving the information on a user-initiatedinput event from the client device.

In Example 75, the subject matter of any one or more of Examples 73-74optionally include wherein the operations further comprise: encoding thedata describing contents of the defined portion of the graphical displayof the display screen to the client device over the network using avideo codec.

In Example 76, the subject matter of any one or more of Examples 73-75optionally include wherein the operations further comprise: receiving achange in the defined portion of the graphical display of the displayscreen to a second defined portion; and sending data describing contentsof the second defined portion of the graphical display of the displayscreen to the client device over the network.

In Example 77, the subject matter of any one or more of Examples 73-76optionally include wherein the operations further comprise: storing thereceived information on the user-initiated input event in an eventqueue; and prior to consuming the modified information on theuser-initiated input event, determining that a largest sequence numberof events in the event queue is less than a sequence in the informationon the user-initiated input event.

In Example 78, the subject matter of any one or more of Examples 73-77optionally include wherein the operations further comprise: sending athumbnail representation of the entire graphical display of the displayscreen to the client device over the network.

What is claimed is:
 1. A computing device comprising: a display screen;an input device; a processor; a memory coupled to the processor andcomprising instructions, when performed by the processor, causes thecomputing device to perform operations comprising: receiving from aserver device, data describing a portion of a graphical display of theserver device; rendering the portion of the graphical display of theserver device on the display screen using the received data; receivingan input via the input device; sending data describing the input to theserver device, the server device processing the input from the inputdevice of the computing device as local input on the server device;receiving an acknowledgement from the server device that the datadescribing the input was received; and responsive to the acknowledgementfrom the server device, rendering the input by the processor over theportion of the graphical display of the server device displayed on thedisplay screen at a position of the input.
 2. The computing device ofclaim 1, wherein the operations comprise: ceasing to render the inputover the portion of the graphical display of the server displayed on thedisplay screen when a determination is made that the server device hasconsumed the input.
 3. The computing device of claim 2, wherein theoperations of ceasing to render the input over the portion of thegraphical display of the server displayed on the display screen when adetermination is made that the server device has consumed the inputcomprises: waiting a predetermined period of time; and upon conclusionof the predetermined period of time, determining that the server devicehas consumed the input; and in response, ceasing to render the inputover the portion of the graphical display of the server displayed on thedisplay screen.
 4. The computing device of claim 3, wherein theoperations comprise: receiving from the server device, second datadescribing the portion of the graphical display on the server device;and rendering the portion of the graphical display of the server deviceon the display screen using the received second data, the renderedportion of the graphical display of the server device including a visualrepresentation of the input that is visible after ceasing to render theinput over the portion of the graphical display.
 5. The computing deviceof claim 1, wherein the operations comprise: determining characteristicsof the input by determining how previously consumed inputs near theposition of the input were displayed by examining the received datadescribing the portion of the graphical display of the server device;and wherein rendering the input by the processor over the portion of thegraphical display of the server device displayed on the display screenat the position of the input comprises rendering the input using thedetermined characteristics.
 6. The computing device of claim 1, whereinthe operations comprise: prior to a receipt of an acknowledgement fromthe server device that the data describing the input was received,rendering, in a first manner, the input by the processor over theportion of the graphical display of the server device displayed on thedisplay screen at the position of the input; receiving anacknowledgement from the server device that the data describing theinput was received; responsive to receipt of the acknowledgement fromthe server device, re-rendering in a second manner, the input by theprocessor over the portion of the graphical display of the server devicedisplayed on the display screen at the position of the input; andresponsive to a determination that the server device has consumed theinput, ceasing to render the input over the portion of the graphicaldisplay, the first and second manners being visually distinct.
 7. Thecomputing device of claim 1, wherein the operations comprise: receivingan input via the input device to change the portion of the graphicaldisplay received from the server device to a second portion; receivingfrom the server device second data describing the second portion of thegraphical display of the server device, the second portion beingdifferent than the portion; and rendering the second portion of thegraphical display of the server device on the display screen using thesecond received data.
 8. The computing device of claim 7, wherein theinput to change the portion of the graphical display received from theserver device to the second portion comprises an input directed to anavigation display selecting the second portion.
 9. The computing deviceof claim 8, wherein the operations comprise: sending a command to theserver device to change to the second portion, the command comprises adelta x and a delta y value describing a difference between the portionand the second portion.
 10. The computing device of claim 8, wherein thenavigation display is overlaid on top of the rendered received portionof the graphical display of the server device On the display screen. 11.A computing device comprising: a display screen; an input device; aprocessor; a memory coupled to the processor and comprisinginstructions, when performed by the processor, causes the computingdevice to perform operations comprising: receiving from a server device,data describing a first portion of a graphical display on the serverdevice, the first portion being less than a complete graphical displayon the server device; rendering the first portion of the graphicaldisplay of the server device on the display screen using the receiveddata describing the first portion; receiving an input in the inputdevice; sending data describing the input to the server device, theserver device processing the input from the input device of thecomputing device as local input on the server device; receiving an inputvia the input device from a graphical user interface (GUI) elementdisplayed on the display screen to change the first portion of thegraphical display received from the server device to a second portion;transmitting a command to the server device to change the first portionof the graphical display received from the server device to the secondportion; receiving from the server device second data describing thesecond portion of the graphical display of the server device, the secondportion being less than the complete graphical display on the serverdevice and being different than the first portion; and rendering thesecond portion of the graphical display of the server device on thedisplay screen using the received second data.
 12. The computing deviceof claim 11, wherein the operations further comprise: receiving anacknowledgement from the server device that the data describing theinput was received; and responsive to the acknowledgement from theserver device, rendering the input by the processor over the firstportion of the graphical display of the server device displayed on thedisplay screen at a position of the input.
 13. The computing device ofclaim 12, wherein the operations comprise: ceasing to render the inputover the first portion of the graphical display of the server displayedon the display screen when a determination is made that the serverdevice has consumed the input.
 14. The computing device of claim 13,wherein the operations of ceasing to render the input over the firstportion of the graphical display of the server displayed on the displayscreen when a determination is made that the server device has consumedthe input comprises: waiting a predetermined period of time; and uponconclusion of the predetermined period of time, determining that theserver device has consumed the input; and in response, ceasing to renderthe input over the first portion of the graphical display of the serverdisplayed on the display screen.
 15. The computing device of claim 14,wherein the operations comprise: receiving from the server device,second data describing the first portion of the graphical display on theserver device; and rendering the first portion of the graphical displayof the server device on the display screen using the received seconddata, the rendered first portion of the graphical display of the serverdevice including a visual representation of the input that is visibleafter ceasing to render the input over the first portion of thegraphical display.
 16. The computing device of claim 11, wherein theoperations comprise: determining characteristics of the input bydetermining how previously consumed inputs near the coordinates of theinput were displayed by examining the received data describing the firstportion of the graphical display of the server device; and whereinrendering the input by the processor over the first portion of thegraphical display of the server device displayed on the display screenat the position of the input comprises rendering the input using thedetermined characteristics.
 17. The computing device of claim 11,wherein the operations comprise: prior to a receipt of anacknowledgement from the server device that the data describing theinput was received, rendering, in a first manner, the input by theprocessor over the first portion of the graphical display of the serverdevice displayed on the display screen at the position of the input;receiving an acknowledgement from the server device that the datadescribing the input was received; responsive to receipt of theacknowledgement from the server device, re-rendering in a second manner,the input by the processor over the first portion of the graphicaldisplay of the server device displayed on the display screen at theposition of the input; and responsive to a determination that the serverdevice has consumed the input, ceasing to render the input over thefirst portion of the graphical display, the first and second mannersbeing visually distinct.
 18. The computing device of claim 11, whereinthe operations comprise: wherein the GUI element comprises a navigationdisplay showing a smaller version of the complete graphical display onthe server device and an indication of the first portion of thegraphical display on the server device in relation to the completegraphical display on the server device; and wherein the input to changethe first portion of the graphical display received from the serverdevice to the second portion comprises an input directed to thenavigation display selecting the second portion.
 19. The computingdevice of claim 18, wherein the command transmitted to the server devicecomprises a delta x and a delta y value describing a difference betweenthe portion and the second portion.
 20. The computing device of claim18, wherein the navigation display is overlaid on top of the renderedreceived portion of the graphical display of the server device on thedisplay screen.